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ABSTRACT 

To  evaluate  the  Navy's  training  under  instructional  systems 
development  (ISD)  procedures  the  annual,  force-level,  NATOPS 
examination  scores  of  pilots  in  operational  squadrons  who  had 
been  trained  under  the  recently  installed  ISD  program  at  VP-31 
were  compared  with  those  who  had  not  been  trained  under  ISD. 
Scores  from  only  the  open-  and  closed-book  written  examina- 
tions, which  pertain  to  normal  and  emergency  procedures  and 
aircraft  systems  knowledge,  were  used  since  their  content  re- 
flected most  closely  the  course  objectives  of  the  nontactical 
pilot  training  provided  by  VP-31.   Pilot  rank,  designation, 
time-in-squadron,  and  squadron  location  were  used  as  controls. 
The  type  of  training  made  no  difference  in  open-book  scores. 
For  closed-book  scores,  ISD  training  resulted  in  a  marginal 
but  consistently  negative  difference  as  a  main  effect  and  in 
a  significant  negative  difference  when  interactive  with 
squadron  location.   At  one  location  ISD-trained  pilots  scored 
significantly  lower  while  there  was  no  difference  among 
pilots  at  another  location.   The  findings  are  discussed  with- 
in the  context  of  the  study  and  with  respect  to  training  and 
ISD  in  general. 
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I.   INTRODUCTION 

A.  BACKGROUND 

In  support  of  its  aviation  mission,  the  U.S.  Navy  operates 
training  facilities  located  throughout  the  country  to  prepare 
both  officers  and  enlisted  personnel  for  jobs  within  the 
aviation  community.    One  such  facility  is  Patrol  Squadron 
Thirty  One  (VP-31) ,  which  is  the  Navy's  west  coast  P-3  Orion 
Replacement  Air  Group  (RAG) .   VP-31  is  responsible  for, 
inter  alia,  the  training  of  maintenance  and  flight  crew 
personnel  destined  for  operational  P-3  squadrons  located  in 
California  and  Hawaii. 

Prior  to  the  introduction  of  modern  instructional  systems 
design  (ISD)  methods,  VP-31  conducted  pilot  training  utiliz- 
ing traditional  methods  and  media  consisting  primarily  of 
classroom  lectures,  examinations  of  comprehension,  etc. 
Course  objectives  tended  to  be  more  content  oriented  --"famil- 
iarize the  student  with  performance  charts  contained  in 
chapters  11  and  12" —  rather  than  behaviorally  oriented. 
Inherent  in  these  traditional  methods  was  the  problem  of 
measuring  the  effectiveness  of  training. 

B.  INSTRUCTIONAL  SYSTEMS  DEVELOPMENT  (ISD) 

Following  World  War  II  and  the  establishment  of  systems 
analysis  as  a  scientific  discipline,  psychologists  and 


educators  recognized  the  possible  benefits  of  applying 
systems  technology  to  the  field  of  instruction.   These 
pioneers  of  modern  instructional  technology,  who  came  not 
only  from  academia  but  also  from  quasi-governmental  agencies 
such  as  RAND  and  HUMRRO,  saw  great  potential  in  the  use  of 
techniques  such  as  modelling  to  increase  the  precision, 
objectivity,  and  creativity  of  the  judgment  they  exercised 
in  solving  the  problems  of  course  design  [Montemerlo,  1979], 
Perhaps  the  most  significant  contribution  to  the  field  of 
instruction  by  training  technologists  has  been  a  demanding 
emphasis  upon  the  precise  definition  of  learning  objectives 
and  goal-oriented  performance  evaluation.   Enabling  objec- 
tives, final  objectives,  and  course  objectives  form  a  hierarchy 
which  serves  to,  among  other  things,  provide  a  clear  pathway 
for  both  students  and  instructors,  and  to  answer  the  question — 
"precisely  what  will  a  student  be  able  to  do  after  success- 
fully completing  a  course  of  instruction?"   Goal-oriented 
performance  evaluation  provides  a  method  to  measure  accurately 
the  effectiveness  of  training  programs  that  are  oriented 
toward  producing  graduates  capable  of  actually  doing  the 
job. 

Instruction  can  be  divided  into  five  categories  represen- 
ting learned  capabilities,  i.e.,  verbal  information,  cog- 
nitive strategies,  intellectual  skills,  motor  skills,  and 
attitudes  [Gagne  &  Briggs,  1979].   Associated  with  each 
variety  of  learned  capability  are  conditions  for  learning. 
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Traditional  instructional  methods  have  always  emphasized 
external  conditions  for  learning  such  as  repetition  and  stim- 
ulous-response  reinforcement.   Modern  theories  of  learning 
focus  also  upon  the  internal  conditions  for  learning  such  as 
prior  learning,  recall,  and  various  levels  of  memory  [Gagne 
&  Briggs,  1979] . 

Taxonomies  such  as  the  five  types  of  learning  described 
by  Gagne  serve  to  group  similar  objectives  in  a  course  design 
and  provide  for  mutually  applicable  learning  conditions. 
Additionally,  grouping  facilitates  the  employment  of  like 
methods  of  evaluation.   Performance  measures  of  a  motor  skill 
such  as  the  manipulation  of  aircraft  controls  requires  a 
student  to  perform  bodily  movements  involving  muscular  activity 
The  performance  measure  appropriate  to  the  learning  of  verbal 
information  requires  a  student  to  state  the  relationship 
between  two  or  more  named  objects  or  events  either  orally  or 
in  writing. 

Attitudes  are  complex  states  of  human  beings  relating  to 
things,  people,  or  events  which  can  be  described  in  terms  of 
either  their  cognitive  or  affective  aspects.   Measurement  of 
attitudes  is  most  logically  suited  to  the  affective  aspect 
and  therefore  centers  upon  observing  a  person's  chosen  al- 
ternative course  of  action  from  a  selection  of  options. 

Intellectual  skills  form  a  hierarchy  which  starts  with 
simple  discriminations  and  concrete  concepts.   Simple  rules, 
which  include  defined  concepts,  follow  discriminations  and 
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lead  to  problem  solving  through  higher-order  rules.   Perfor- 
mance measures  of  intellectual  skills  range  from  the  basic 
act  of  distinguishing  objects  to  the  formulation  and  employ- 
ment of  complex  rules  in  order  to  achieve  solutions  to 
unique  problems  which  are  new  to  the  student. 

Cognitive  strategies  are  special  forms  of  intellectual 
skills.   These  special  skills  actually  represent  processes 
by  which  learners  modify  their  ways  of  attending,  learning, 
remembering,  and  thinking.   The  salient  difference  between 
intellectual  skills  in  general  and  cognitive  strategies  is 
the  object  of  the  skill.   Cognitive  strategies  focus  upon 
the  learner's  own  thought  process  whereas  intellectual  skills 
relate  to  objects  and  events  [Gagne  &  Briggs,  1979].   The 
performance  measure  of  a  cognitive  strategy  requires  the 
novel  solution  to  a  problem  where  familiar  solutions  are 
not  suitable. 

Goal-oriented  performance  objectives  are  requisite 
measures  of  learning  success  in  ISD  programs.   However,  the 
question  of —  "performance  quality"  must  be  considered. 
Objective-referenced  examinations  employing  a  criterion- 
referenced  interpretation  provide  the  most  acceptable  pro- 
cedure for  a  variety  of  reasons.   The  validity  of  an  objective- 
referenced  test  is  directly  related  to  its  congruence  with 
the  course  objective.   Reliability  of  such  a  test  is  derived 
from  the  degree  of  consistency  of  repeated  examinations. 
In  an  objective-referenced  test  of  intellectual  skills, 
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motor  skills,  or  verbal  information,  mastery  is  a  relevant 
concept  and  error-free  performance  can  be  achieved.   However, 
cognitive  strategies  and  attitudes  are  not  suited  to  the 
mastery  criterion  because  of  their  lack  of  an  absolute 
measure  [Gagne  &  Briggs,  1979]. 

Group  instruction  can  be  regarded  in  three  separate 
categories  which  differ  in  the  amount  of  precision  afforded 
to  the  management  of  instructional  events  by  the  instructor. 
In  a  two-person  tutorial  situation,  the  instructor  is  pro- 
vided with  the  most  control  over  instructional  events.   A 
small  group  is  defined  as  three  to  eight  people  and  is 
dependent  upon  group  interaction  to  provide  instructional 
events  such  as  feedback,  recall,  etc.  [Gagne  &  Briggs,  1979]. 
Large  groups  of  more  than  15  people  are  characterized  by 
weak  control  over  instructional  events  by  the  instructor. 
Accordingly,  in  a  large  group,  learning  is  highly  dependent 
upon  the  learner's  own  strategies  of  self  instruction.   To 
counter  the  inherent  weak  control  of  learning  in  large 
groups,  the  mastery  concept  for  enabling  objectives  serves 
to  diagnose  learning  performance  early  on  and  facilitate 
attainment  of  the  course  objectives  in  ISD  programs. 

Individualized  instruction  is  designed  in  the  same 
fashion  that  individual  lessons  are  designed  for  group  in- 
struction.  Objective  hierarchies,  learning  events,  and 
conditions  for  learning  all  apply  to  the  design  of  modules 
suited  to  individualized  instruction.   The  distinction  lies 
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in  the  delivery  system  employed  for  individualized  instruction 
Modules  are  normally  self-paced  and  the  materials  do  most  of 
the  teaching.   More  frequent  feedback  and  progress  checks  are 
found  in  programs  of  individualized  instruction.   Under  an 
ISD  system,  students  are  typically  provided  a  series  of 
modules  which  each  contain: 

1.  A  performance  objective 

2.  A  set  of  materials  and  learning  activities 

3.  A  method  of  self -evaluation  to  determine  mastery 

4.  A  provision  for  verification  of  learning  to  the 
instructor  [Gagne  &  Briggs,  1979] 

As  an  organization  responsible  for  a  myriad  of  training 
programs  ranging  in  subject  matter  from  remedial  reading  to 
the  conduct  of  war  on  a  global  basis,  it  was  not  at  all 
surprising  that  the  U.S.  military  establishment  became  in- 
terested and  involved  in  the  methods  of  ISD  as  it  gained  in 
credibility.   With  its  basis  in  scientific  methods  and  logical 
appeal,  ISD  became  an  attractive  alternative  to  traditional 
training  methods  which  offered  the  unique  advantage  of  a 
capability  to  measure  the  effectiveness  of  a  training  program 
in  terms  of  job-related  performance. 

Today,  ISD  is  an  integral  part  of  the  training  conducted 

by  all  four  services.   ISD  has  been  employed  by  the  U.S. 

military  to  ensure  that: 

1.   Courses  are  designed  to  teach  only  those  tasks 
which,  based  upon  objective  field  research  and 
analysis  of  the  tasks  needed  to  be  performed, 
the  graduate  will  use  and  which  can  be  taught 
in  a  formal  training  course. 
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2.  Tests,  the  requisite  for  graduation,  are  accurate 
indicators  of  the  ability  to  perform  the  required 
task  [Department  of  Defense,  1980  p.  XI-7] . 

The  five  phase  ISD  program,  — i.e.,  analyze,  design, 
develop,  implement,  and  control —  and  associated  phase  steps 
is  presented  in  Figure  1. 

The  Navy  is  committed  to  the  systematic  design  of  its 
training  courses  [Chief  of  Naval  Education  and  Training  (CNET) , 
1975;  Scanland,  1974],    The  Interservice  Procedures  for  In- 
structional Systems  Development  (Chief  of  Naval  Education 
and  Training)  have  been  summarized  by  the  Navy  for  its  own 
use  in  a  procedures  manual  entitled  Procedures  for  Instruction- 
al Systems  Development  [CNET,  1978].   AF  Pamphlet  50-58  [De- 
partment of  the  Air  Force,  July  197 S]  describes  the  Air  Force 
ISD  procedures  and  AF  Manual  50-2  [Department  of  the  Air 
Force,  July  197  5]  promulgates  the  following  policy  regarding 
the  utilization  of  ISD. 

1.  Apply  ISD  to  all  new  instructional  systems 

2.  Selectively  apply  ISD  to  existing  instructional 
systems  where  economically  feasible 

The  extensive  use  of  ISD  by  the  military  has  prompted 
periodic  review  of  training  programs  [Brock,  1977;  Logicon 
Incorporated,  1978]  .   These  studies  emphasize  the  importance 
of  job-related  learning  objectives  as  a  prerequisite  to 
successful  ISD  programs.   If  lists  of  learning  objectives 
are  provided  for  students,  it  seems  natural,  however,  that 
their  efforts  in  learning  will  be  generally  limited  to 
those  items  on  the  list.   Consequently,  ISD  program  design- 
ers must  leave  no  stone  unturned  in  an  exhaustive  search 
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Figure  1.   The  Instructional  Systems  Development  Process 
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to  identify  all  relevant  items  in  a  task  analysis.   Under 
ISD,  it  is  naive  to  assume  that  the  learning  of  important  but 
easily  overlooked  items,  such  as  a  desired  attitude  or  a 
knowledge  of  the  relationship  of  components  to  a  total  system, 
will  take  place  in  the  absence  of  specifically  related 
learning  objectives. 

Montemerlo  [1979]  cautions  designers  of  instructional 
systems  against  a  potential  problem  associated  with  the 
atomization  approach  of  ISD  manuals.   He  alleges  that  students 
inundated  with  a  torrent  of  modularized  learning  objectives 
may  miss  an  overall  goal.   Atomistic  descriptions  tend  to 
feed  the  learner  a  large  number  of  new  terms,  definitions, 
and  relationships  faster  than  he  can  assimilate  them  [Monte- 
merlo, 1979] .   The  spiral  approach  to  learning,  which  is 
the  basis  for  most  apprenticeship  programs,  focuses  upon 
progressively  more  complex  hands-on  practice  interspersed 
with  limited  book-learning.   It  is  because  of  channel  capacity, 
which  is  the  amount  of  information  which  a  person  can  process 
before  becoming  confused,  that  Montemerlo  [1979]  asserts 
spiral  learning  works  better  than  atomistic  descriptions. 

Training  is  an  investment  in  human  capital  and  represents 
an  organizational  asset.   Associated  with  a  training  invest- 
ment in  human  capital  is  an  allocation  of  resources,  such 
as  time  and  money.   The  ISD  method  of  instruction  is  extremely 
costly  in  terms  of  time  required  to  develop  and  implement  a 
program.   Lexicon  Incorporated,  [1978]  have  tentatively 
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suggested  an  excess  of  five  years  is  required,  under  ideal 
conditions,  for  the  conversion  of  typical  flying  programs. 
However,  ISD  programs,  with  self-instruction,  offer  the 
potential  of  reducing  the  manhours  spent  by  instructors  in 
teaching.  In  order  to  determine  accurately  the  total  cost 
of  an  ISD  application,  the  extensive  costs  related  to 
development  of  the  program  must  not  be  overlooked  as  these 
front-end  costs  are  significant. 

The  evaluation  of  a  learning  program  is  typically  cen- 
tered upon  three  questions: 

1.  To  what  extent  have  the  stated  objectives  of 
instruction  been  met? 

2.  In  what  ways  and  to  what  degree  is  it  better  than 
the  unit  (of  instruction)  it  will  supplant? 

3.  What  additional, possibly  unanticipated,  effects 
has  it  had  and  to  what  extent  are  these  better  or 
worse  than  the  supplanted  unit?  [Gagne  &  Briggs, 
1979] 

Formative  evaluation  is  normally  conducted  during  the 

development  of  a  new  system  and  serves  to  enhance  the  final 

product  through  modification  or  revision.   By  formative 

evaluation,  quality  in  the  design  of  a  program  can  be  ensured 

Summative  evaluation,  on  the  other  hand,  is  performed  upon 

an  existing  course  of  instruction  and  normally  serves  to 

compare  a  new  system  to  an  "old  one."   During  a  summative 

evaluation,  variables  such  as  student  aptitude  for  learning, 

the  process  of  instruction,  and  learning  support  items  must 

be  controlled  if  a  truly  meaningful  and  unbiased  evaluation 

is  to  result. 
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The  training  of  Naval  Aviators  for  duty  as  pilots  and 
copilots  in  P-3  aircraft  involves  primarily  the  learning 
taxonomies  of  motor  skills, verbal  information,  and  intel- 
lectual skills.   However,  positive  attitudes  towards  things 
such  as  safety  and  furthering  aviator  skills  should  also  be 
considered  germane  to  a  total  training  program.   There  is 
an  emerging  acceptance  of  ISD  as  a  total  systems  approach 
rather  than  a  series  of  steps  to  be  accomplished  on  some 
segment  or  portion  of  flying  training.   This  acceptance  is 
primarily  responsible  for  the  fact  that  ISD  teams  are 
beginning  to  return  to  the  task  analysis  in  order  to 
strengthen  it  by  means  of  a  systematic,  rather  than  a 
cursory  or  short-cut  approach  [Lexicon  Incorporated,  1978, 
p.  58]. 

The  purpose  of  this  study  was  to  examine  the  effective- 
ness of  the  application  of  ISD  to  VP-31  pilot  training  as 
measured  by  scores  achieved  upon  annually  administered, 
force-level,  NATOPS  examinations.   Another  study  currently 
underway  [Fadness,  1980],  has  been  designed  to  examine  the 
application  of  ISD  to  the  training  of  P-3  acoustic  sensor 
operators  at  VP-31. 
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II.   METHOD 

A.   APPROACH 

In  order  to  evaluate  the  effectiveness  of  pilot  training 
at  VP-31  under  a  new  ISD  program,  a  sample  of  both  ISD  and 
non-ISD  trained  graduates  was  required.   Under  ideal  circum- 
stances, an  experimental  group  (ISD)  and  an  identical  control 
group  (non-ISD)  would  be  trained  in  parallel  under  conditions 
as  identical  as  possible.   Following  graduation,  the  training 
received  by  both  groups  would  be  evaluated  using  identical, 
job-related  performance  measures.   Further,  retention  of 
training  could  be  measured  at  a  later  time. 

Under  the  constraints  that  this  study  experienced,  the 
experimental  and  control  groups  had  to  be  reconstructed  from 
individuals  already  in  operational  squadrons  located  at  two 
west-coast  P-3  bases  who  had  received  training  under  one 
condition  or  the  other.   Since  the  training  received  by  these 
individuals  had  occurred  over  a  considerable  period  of  time, 
the  individuals  varied  extensively  in  both  pre-  and  post- 
training  experiences.   Accordingly,  demographic  data  were 
collected  to  control  for  these  differences  statistically. 
Scores  achieved  upon  annually  administered  force  level 
standardization  examinations  were  selected  as  the  dependent 
variable  because  of  the  relatively  constant  examination 
content  over  the  time  period  examined. 
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B.   DESCRIPTION  OF  ANNUALLY  ADMINISTERED  STANDARDIZATION 
EXAMINATIONS 

Standardization  of  normal  and  emergency  operating  proce- 
dures for  crewmembers  of  Naval  aircraft  is  provided  by  a 
program  known  as  Naval  Air  Training  and  Operating  Procedures 
Standardization  (NATOPS) .   Included  within  the  NATOPS  program 
for  P-3  Orion  pilots  is  a  squadron-level,  initial  certifica- 
tion to  serve  in  the  capacity  of  pilot-in-command  and  subsequent 
annual  recertif ications.   Each  certification  process  consists 
of  two  written  examinations  (one  open  book  and  one  closed 
book)  and  an  inflight  check  of  piloting  skills.   Each  phase 
of  this  examination  process  is  designed  to  evaluate  stated 
skill  areas.   Open-book  examinations  contain  24  multiple- 
choice  questions  relating  primarily  to  performance  charts 
located  within  the  NATOPS  manual.   Pilots  are  required  to 
compute  various  performance  values  from  charts  and  select 
the  correct  answer  to  a  written  question  from  four  choices. 
Closed-book  examinations  contain  40  multiple  choice  questions 
relating  to  normal  and  emergency  procedures  and  aircraft 
systems.   Pilots  are  required  to  respond  to  a  written  question 
by  selecting  one  of  four  possible  answers.   Flight  checks, 
which  last  approximately  three  hours,  are  designed  to  evaluate 
both  knowledge  of  procedures  and  systems  (verbal  answers  to 
verbal  questions),  and  piloting  skills.   The  content  of  written 
examinations  must  follow  a  distribution  of  subject  matter 
— number  of  questions  per  subject —  specified  by  the  force- 


21 


level  evaluation  team.   To  assist  in  examination  construction, 
the  force-level  NATOPS  evaluation  team  maintains  a  sizeable 
bank  of  sample  questions  for  each  subject  area.   Objective 
performance  criteria  for  flight  checks  is  provided  within  the 
NATOPS  program  to  aid  in  standardization. 

In  addition  to  squadron  level  NATOPS  examinations,  force 
level  NATOPS  examinations  are  administered  to  each  operational 
squadron  on  an  annual  basis.   Force-level  examinations  consist 
of  two  written  examinations  (one  open  book  and  one  closed 
book)  for  all  crewmembers  and  flight  checks  for  only  three 
selected  personnel  per  crew  position.   Force-level  evaluations 
are  intended  to  be  similar  in  content  to  squadron-level 
evaluations.   Crewmembers  certified  by  the  NATOPS  force-level 
evaluator  are  awarded  squadron  NATOPS  instructor  status  and 
are  authorized  to  conduct  squadron-level  NATOPS  check  flights 
for  their  crew  position.   The  results  of  force-level  NATOPS 
unit  evaluations  for  west-coast  (California  and  Hawaii)  P-3 
squadrons  are  maintained  by  the  COMNAVAIRPAC  P-3  NATOPS 
evaluation  team.   Force  level  NATOPS  written  examination 
scores  achieved  by  P-3  pilots  between  1978  and  1980  were 
made  available  for  this  study  by  the  Commanding  Officer, 
Patrol  Squadron  Thirty  One,  and  the  COMNAVAIRPAC  NATOPS 
pilot  evaluator.   It  was  agreed  that  the  names  of  individual 
pilots  and  their  respective  squadrons  would  remain  anonymous 
throughout  the  analysis. 


22 


C.   SAMPLE  DESCRIPTION 

Between  January  1978  and  May  1980,  31  west  coast,  opera- 
tional P-3  squadrons  received  force  level,  NATOPS,  unit 
evaluations.   In  the  Spring  of  1979,  graduates  of  VP-31's  new 
ISD  training  program  began  reporting  to  west-coast  P-3  squadrons 
and  participating  in  scheduled,  force-level,  NATOPS  examina- 
tions.  Three  hundred  eighty-two  pilots  from  10  squadrons 
located  in  California  and  Hawaii  were  selected  as  the  initial 
sample.   Composition  of  the  initial  sample  was  based  upon  two 
primary  factors: 

1.  Obtaining  a  maximum  number  of  ISD-trained  pilots 

2.  The  deployment  status  of  squadrons  which  determined 
their  ability  to  provide  demographic  data  on  sample 
pilots 

Demographic  data  regarding  the  sample  pilots  were  provided  by 

squadrons  through  crew  lists  applicable  to  the  date  of  the 

unit  evaluation. 

During  initial  analysis  it  was  not  surprising  to  discover 

that  because  of  the  January  1979  implementation  date  of  ISD 

at  VP-31,  sample  pilots  trained  under  the  ISD  condition  had 

generally  been  onboard  their  assigned  operational  squadrons 

only  a  few  months.   Accordingly,  the  sample  was  reduced  to 

those  pilots  with  12  or  fewer  months  experience  onboard  their 

assigned  squadrons.   This  reduction  provided  for  a  total  of 

123  total  pilots  of  which  61  were  trained  under  the  ISD 

program  (Tables  1  and  2) . 
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Table  1 

Sample  of  Pilots  Used  in  the  Study  by  Rank,  Months 
in  the  Squadron,  Type  of  Training  and  Location* 


RS 

iNK 

MONTHS 

TOTAL 

1-4 

5- 

•8 

9-12 

Location  A 

0-5 

2    (    1) 

0 

(    1) 

1    (    1) 

3     (    3) 

0-4 

2    (    5) 

4 

(    2) 

2     (    2) 

8     (    9) 

0-3 

1    (    1) 

2 

(    1) 

2     (    2) 

5     (    4) 

0-2 

6    (12) 

8 

(    6) 

14     (    2) 

28     (20) 

0-1 

1    (    2) 

2 

(    0) 

0     (    0) 

3     (    2) 

Sub- 

•total 

12     (21) 

16 

(10) 

19     (    7) 

47     (38) 

0-5 
0-4 
0-3 
0-2 
0-1 
Sub- total 

Total 


Location   B 

0 

(    0) 

1    (    0) 

0 

(    3) 

1    (    3) 

0 

(    1) 

1    (    0) 

1 

(    7) 

5    (    7) 

0 

(    1) 

0    (    0) 

1 

(12) 

8     (10) 

13  (33) 


24  (20) 


0  (  1) 

0  (  0) 

1  (  0) 

5  (  0) 
0  (  0) 

6  (  1) 

25  (  8) 


1  (  1) 

1  (  6) 

2  (  1) 
11  (14) 

0  (  1) 

15  (23) 

62  (61) 


*  Numbers  in  parenthesis  represent  those  who  were  trained 
under  the  ISD  condition 
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Table  2 

Sample  of  Pilots  Used  in  the  Study  by  Rank, 
Designation,  Type  of  Training,  and  Location* 


RANK 


NOP 


DESIGNATION 


PP3P 


PP2P 


PPC 


TOTAL 


Location 

A 

0-5 

1 

(  0) 

0 

(  0) 

0 

(1) 

2 

(2) 

3 

(  3) 

0-4 

3 

(  3) 

0 

(  0) 

1 

(2) 

4 

(4) 

8 

(  9) 

0-3 

3 

(  1) 

2 

(  3) 

0 

(0) 

0 

(0) 

5 

(  4) 

0-2 

8 

(14) 

18 

(  6) 

2 

(0) 

0 

(0) 

28 

(20) 

0-1 

2 

(  1) 

1 

(  1) 

0 

(0) 

0 

(0) 

3 

(  2) 

Sub-total 

17 

(19) 

21 

(10) 

3 

(3) 

6 

(6) 

47 

(38) 

Location 

B 

0-5 

0 

(  0) 

0 

( 

0) 

0 

(0) 

1 

(1) 

0-4 

1 

(  5) 

0 

( 

0) 

0 

(0) 

0 

(1) 

0-3 

0 

(  1) 

2 

( 

0) 

0 

(0) 

0 

(0) 

0-2 

5 

(10) 

6 

( 

4) 

0 

(0) 

0 

(0) 

0-1 

0 

(  1) 

0 

( 

0) 

0 

(0) 

0 

(0) 

Sub- total 

6 

(17) 

8 

( 

4) 

0 

(0) 

1 

(2) 

1  (  1) 

1  (  6) 

2  (  1) 
11  (14) 

0  (  1) 

15  (23) 


Total 


23  (36)    29  (14) 


3  (3) 


7  (8) 


62  (61) 


Numbers  in  parenthesis  represent  those  who  were  trained  under 
the  ISD  Condition 
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D.  DEPENDENT  VARIABLES 

Force-level  NATOPS  written  examination  scores  achieved 
by  the  123  pilots  during  unit  evaluations  were  selected  as 
the  dependent  variable  on  which  to  evaluate  phase  A  and  B 
pilot  training  conducted  at  VP-31.   The  suitability  of  force- 
level  written  examinations  as  a  training  effectiveness 
measure  is  established  by  the  close  agreement  between  the 
course  objectives  of  phase  A  and  B  pilot  training  and  the 
subject  matter  specified  for  testing  in  the  P-3  NATOPS 
program.   During  phase  A  and  B,  pilot  training  subject  matter 
consists  of  normal  and  emergency  procedures  and  the  operation 
of  aircraft  systems.   It  is  during  phase  C  and  D,  which  have 
been  purposely  excluded  from  this  study,  that  training  in  the 
various  aspects  of  ASW  and  tactics  is  conducted. 

E.  INDEPENDENT  VARIABLES 

Various  demographic  variables  in  addition  to  the  type  of 
training  received  were  considered,  and  the  following  five  were 
ultimately  selected  (See  Table  3)  . 

1.   Instructional  Systems  Development  (ISP) 

The  Instructional  Systems  Development  variable  iden- 
tifies the  type  of  training  that  a  pilot  within  the  sample 
received  while  attending  VP-31.   Those  pilots  who  were  trained 
under  the  new  system  implemented  in  January  of  1979  were 
coded  1,  and  those  trained  under  the  old  system  were  coded 
0. 
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Table  3 


Definition  of  Variables 


NAME 


VALUE 


LEGEND 


Pilot  Rank 

1 

(RANK) 

2 

3 

4 

5 

Designation 

1 

(DESIG) 

2 

3 

4 

Months 

1 

(DMONTHS) 

2 

Training 

0 

(ISD) 

1 

Squadron 

1 

(DSQD) 

2 

Ensign 

Lieutenant  Junior  grade 

Lieutenant 

Lieutenant  Commander 

Commander 

Patrol  Place  Commander  (PPC) 

Patrol  Plane  Second  Pilot  (PP2P) 

Patrol  Plane  Third  Pilot  (PP3P) 

Non-Designated  Pilot  (NOP) 

1-4  Months  Onboard  Squadron  at  Time 
of  Test 

5-8  Months  Onboard  Squadron  at  Time 
of  Test 

9-12  Months  Onboard  Squadron  at  Time 
of  Test 

Non-ISD-Trained  Pilot 

ISD-Trained  Pilot 

All  Squadrons  Located  at  Location  A 

All  Squadrons  Located  at  Location  B 
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2.  Pilot  Rank 

The  rank  of  all  sample  pilots  at  the  time  they  took 
the  annual  NATOPS  examination  was  coded  from  1  to  5  represent- 
ing the  prescribed  military  structure  of  0-1  through  0-5. 

3 .  Designation 

Pilots  in  the  P-3  community  are  grouped  into  four 
general  categories  which  are  determined  by  their  qualification 
as  pilots.   Nonrated  pilots  (NOP)  are  those  pilots  who  have 
not  yet  qualified.   Patrol  Plane  Third  Pilots  (PP3P)  have 
successfully  completed  the  first  stage  of  the  P-3  pilot 
Personal  Qualifications  Standards  (PQS)  training  but  have 
not  yet  completed  a  NATOPS  check.   Patrol  Plane  Second  Pilots 
(PP2P)  have  successfully  completed  the  second  stage  of  PQS 
training  and  a  pilot  NATOPS  check.   A  PP2P  is  considered 
"safe  for  solo"  and  may  serve  as  pilot-in-command  during  a 
nontactical  mission.   Patrol  Plane  Commanders  (PPC)  have 
successfully  completed  the  required  PQS  and  the  training 
directed  by  the  Commander/  Patrol  Wings  Pacific  (CPWP) . 
Patrol  Plane  Commanders  are  authorized  to  serve  as  pilot-in- 
command  of  tactical  missions. 

4 .  Months 

The  months  variable  represents  the  number  of  months 
that  a  pilot  in  the  sample  was  onboard  his  squadron  at  the 
time  of  the  force-level  NATOPS  examination.   Although  this 
variable  ranged  from  1  to  36,  the  sample,  as  stated,  was 
restricted  to  those  pilots  with  from  1  to  12  months  onboard 
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a  squadron  when  the  examination  was  administered.   This 
restriction  reduced  the  sample  size  to  123  but  provided  for 
a  more  representative  distribution  of  the  ISD  variable  due 
to  the  newness  of  ISD  at  VP-31.   To  facilitate  analysis, 
aggregation  of  the  months  variable  resulted  in  three  groups: 
1-4  months,  5-8  months,  and  9-12  months. 
5.   Squadron 

The  squadron  variable  served  to  distinguish  the  vari- 
ous squadrons  to  which  west  coast  P-3  pilots  are  assigned 
subsequent  to  VP-31  training.   Aggregation  of  the  squadron 
variable  resulted  in  two  groups  representing  the  two  bases 
at  which  squadrons  are  located. 

In  the  case  of  all  variables,  aggregation  was  identi- 
fied by  a  preceding  letter  D. 

F.   ANALYSIS  OF  THE  DATA 

Analysis  of  the  data  was  conducted  at  three  levels  to 
investigate  all  potential  aspects  of  the  problem.   Initial 
analysis  focused  upon  the  distribution  of  open-  and  closed- 
book  scores  by  demographic  variables.   Additionally,  tests  of 
independence  between  the  distribution  of  the  subjects  on  the 
ISD  variable  and  each  of  the  demographic  variables  were  com- 
puted and  examined  using  chi-square.   Utilizing  further 
aggregated  variables  (see  Results  section) ,a2x2x2x3 
factorial  analysis  of  variance  of  open-  and  closed-book 
examination  scores  was  conducted.   This  investigation  utilized 
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a  Statistical  Package  for  the  Social  Sciences  [Nie,  Hull, 
Jenkins,  Stienbrenner  &  Bent,  1978]  computer  program  and  was 
designed  to  reveal  the  significance  of  four  demographic 
variables  (ISD,  DSQD,  DRANK,  and  DMONTHS)  in  accounting  for 
the  heterogeneity  of  open-  and  closed-book  examination  scores. 
Due  to  missing  DDESIG  data  points  in  the  sample  (Table  2) , 
the  DDESIG  variable  was  excluded  from  the  factorial  analysis 
of  variance  to  permit  the  computation  of  two-way  interactions. 
Further,  in  view  of  the  high  degree  of  dependence  between 
the  DDESIG  and  DRANK  variables  (Chi-square  test  of  indepen- 
dence significant  at  <.001),  exclusion  of  the  DDESIG  variable 
would  not  detract  from  the  analysis.   Three-way  or  higher 
interactions  were  pooled  with  the  error  sum  of  squares.   Ag- 
gregated variables  were  again  used  for  multivariate  analysis 
of  the  variance  among  open-  and  closed-book  examination 
scores  considered  simultaneously.   This  analysis  was  executed 
utilizing  the  Statistical  Analysis  System  [SAS  Institute, 
1979]  computer  program  and  examined  the  effects  of  all  five 
demographic  variables  in  accounting  for  the  variance  of  com- 
bined scores. 

In  addition  to  the  statistical  analyses  described, 
operational  factors  associated  with  the  conduct  of  training 
at  VP-31  were  examined.   Fiscal  year  averages  of  salient 
measures  were  compared  for  FY77,  FY78  and  FY79.   Factors 
such  as  staff  size,  instructor  complement,  student  load,  time 
spent  in  ground  school  during  the  familiarization  phase,  and 
total  syllabus  length  were  considered  germane  to  the  study. 
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III.       RESULTS 

A.  SUMMARY  RESULTS 

Based  upon  the  initially  defined  independent  variables, 
means  and  standard  deviations  for  open-  and  closed-book  ex- 
aminations were  computed  by  category  of  training  received  at 
VP-31  (See  Table  4) . 

B.  AGGREGATED  VARIABLES 

Further  analysis  of  the  problem  was  enhanced  by  aggrega- 
tion of  the  rank  and  designation  variables  to  permit  the  use 
of  multivariate  procedures.   The  five-category  rank  variable 
was  aggregated  into  two  groups:  0-1  through  0-3  and  0-4 
through  0-5.   This  distribution  provided  for  not  only  a 
grouping  of  junior  and  senior  officers,  but  additionally,  it 
provided  an  identification  of  first-  and  second-tour  pilots. 
The  four  level  designation  variable  was  combined  to  achieve 
two  levels  of  designation:  NOP  and  PP3P,  and  PP2P  and  PPC . 
This  division  provided  a  means  to  identify  pilots  who  had 
completed  a  NATOPS  certification  check.   The  squadron  vari- 
able and  months  variable  were  already  aggregated  (See  Table 
3) .   Means  and  standard  deviations  for  the  newly  aggregated 
rank  and  designation  variables  are  presented  in  Table  5. 

Tables  6  and  7  present  a  contingency  analysis  of  the 
training  variable  with  each  of  the  aggregated  demographic 
variables  to  check  for  independence  of  the  training  variable 
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Table  4 

Means  and  Standard  Deviations  of  Annual 
NATOPS  Open  and  Closed  Book  Examinations 
by  Category  of  Training  Received* 


DEMOGRAPHIC 

VALU 

E 

TEST 

OPEN 

CLOSED 

NON- 

NON- 

ISD 

ISD 

ALL 

ISD 

ISD 

ALL 

Pilot  Rank 

1 

3.55 

3.44 

3.50 

3.33 

3.60 

3.47 

.48 

.34 

.38 

.59 

.17 

.41 

(RANK) 

2 

3.69 

3.73 

3.71 

3.42 

3.62 

3.53 

.31 

.25 

.28 

.39 

.27 

.35 

3 

3.60 

3.51 

3.55 

3.32 

3.54 

3.45 

.32 

.53 

.44 

.39 

.30 

.34 

4 

3.85 

3.89 

3.87 

3.74 

3.77 

3.75 

.18 

.17 

.17 

.24 

.30 

.26 

5 

4.00 

3.79 

3.90 

3.68 

3.78 

3.73 

.00 

.16 

.15 

.17 

.26 

.21 

Designation 

1 

3.94 

3.86 

3.90 

3.73 

3.81 

3.77 

.09 

.15 

.12 

.21 

.20 

.21 

(DESIG) 

2 

3.94 

3.94 

3.94 

3.83 

3.70 

3.77 

.10 

.10 

.08 

.15 

.26 

.21 

3 

3.74 

3.75 

3.75 

3.46 

3.59 

3.55 

.28 

.27 

.27 

.33 

.30 

.31 

4 

3.67 

3.60 

3.64 

3.44 

3.64 

3.52 

.32 

.34 

.33 

.42 

.25 

.37 

Months 

1 

3.66 

3.58 

3.63 

3.43 

3.56 

3.47 

.32 

.39 

.34 

.44 

.31 

.41 

(DMONTHS) 

2 

3.83 

3.72 

3.77 

3.58 

3.63 

3.61 

.25 

.28 

.27 

.28 

.29 

.30 

3 

3.81 

3.79 

3.79 

3.63 

3.69 

3.67 

.23 

.24 

.23 

.29 

.25 

.26 
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Table  4  (continued) 


Squadron      1    3.69  3.68  3.69  3.48  3.67  3.59 

.34  .31  .32  .43  .27  .36 

(DSQD)        2    3.80  3.83  3.81  3.53  3.54  3.54 

.20  .22  .20  .30  .26  .29 


TOTAL  3.73    3.72    3.73     3.50     3.64     3.57 

.30     .30     .30      .38      .28      .34 


* 

Means  above  and  standard  deviations  below  in  each  set  of 

data 
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Table  6 

Contingency  Analysis  of  the  Distribution 
of  ISD  and  Non-ISD  Trained  Pilots 
by  Demographic  Variables 


DEMOGRAPHIC 
VARIABLES 


VALUES 


ISD 


NON-     CHI       SIGNIFIANCE 
ISD      SQUARE    OF  CHI  SQUARE 


DRANK 


DSQD 


DDESIG 


DMONTHS 


1 
2 

1 
2 

1 
2 

1 
2 
3 


42 

49 

19 

13 

38 

47 

23 

15 

11 

10 

50 

52 

33 

13 

20 

24 

8 

25 

1.17 


30 


2.03 


00 


17.81 


.15 


.97 


.00 
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Table  7 

Contingency  Analysis  of  the  Distribution 
of  ISD  and  Non-ISD  Trained  Pilots 
by  Aggregated  Months  and  Location 


NON- 

CHI 

SIGNIFICANCE 

LOCATION 

DMONTHS 

ISD 

ISD 

SQUARE 

OF  CHI  SQUARE 

A 

1 

21 

12 

8.52 

.01 

2 

10 

16 

3 

7 

19 

B  1         12      1     11.95  .00 


1 

12 

1 

2 

10 

8 

3 

1 

6 
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distribution.   There  was  a  significant  degree  of  dependence 
in  the  distribution  of  the  ISD  variable  by  DMONTHS  at  both 
squadron  locations  examined.   This  result  was  not  surprising 
since  pilots  trained  under  the  new  ISD  system  were  more 
likely  to  have  fewer  months  onboard  their  squadron  than  non- 
ISD  trained  pilots  at  the  time  of  the  unit  NATOPS  evaluations 
The  adjusted  chi-square  computed  for  the  DMONTHS-ISD  matrix 
was  17.8  and  significant  at  the  .0001  level. 

C.   FACTORIAL  ANALYSIS  OF  VARIANCE  OF  OPEN  BOOK  SCORES 
(TABLE  8) 

The  DRANK,  DSQD,  and  DMONTHS  variables  were  found  to  be 
significant  as  main  effects  in  accounting  for  the  observed 
heterogeneity  among  sample  open-book  NATOPS  examination 
scores.   Additionally,  the  interaction  between  DSQD  and 
DRANK  variables  was  determined  to  be  significant.   Mean 
open-book  examination  scores  for  both  values  of  the  DRANK 
variable  were  demonstrated  to  be  nearly  identical  at  loca- 
tion B  and  quite  different  at  location  A.   Accordingly, 
although  the  DRANK  and  DSQD  variables  were  significant  when 
considered  separately,  the  predictive  power  of  the  DRANK 
variable  was  enhanced  by  knowing  the  location  variable 
(See  Table  9  and  Figure  2) .   It  is  interesting  to  note  that 
in  spite  of  the  significant  degree  of  dependence  in  the 
distribution  of  ISD  by  DMONTHS,  the  two-way  interaction 
between  these  variables  in  the  factorial  analysis  was  not 
significant.   Further,  ISD,  per  se,  was  not  identified  as 
a  significant  effect  in  this  analysis. 
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Table  8 

Factorial  Analysis  of  Variance  of  Open-Book 
NATOPS  Scores  by  Demographic  Variables 


DEMOGRAPHIC  VARIABLE 


D/F" 


F  VALUE    SIGNIFICANCE 
OF  F 


ISD 

DMONTHS 
DRANK 
DSQD 


1 

.577 

.45 

2 

5.075 

.01 

1 

14.151 

.00 

1 

6.002 

.02 

Two  Way  Interactions 


ISD 

DMONTHS 

2 

.072 

.93 

ISD 

DSQD 

1 

.514 

.48 

ISD 

DRANK 

1 

.041 

.84 

DMONTHS 

DSQD 

2 

.992 

.37 

DMONTHS 

DRANK 

2 

.279 

.76 

DSQD 

DRANK 

1 

6.233 

.01 

Total  D/F  =  122 
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Table  9 

Mean  Open-Book  NATOPS  Scores  for  Location  (DSQD) 

by  Rank  (DRANK) 


Value  of  DRANK  Location 

A  B 

1  3.61  3.82 

2  3.9  3.81 
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4.0 


Location  A 
Location  B 


SCORE 


3.0 


2.0 


DRANK 


Figure  2.   Interactive  Effects  of  Rank  and  Location 
on  Open-Book  NATOPS  Examination  Scores 
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D.   FACTORIAL  ANALYSIS  OF  THE  VARIANCE  OF  CLOSED-BOOK 
SCORES  (TABLE  10) 

The  factorial  analysis  of  variance  of  closed-book  scores 
identified  the  DMONTHS  and  DRANK  variables  as  significant 
main  effects.   Although  not  significant  at  the  .05  level, 
the  ISD  variable  was  significant  at  the  .08  level.   Further, 
the  two-way  interactions  of  ISD  with  DSOD  and  DSQD  with 
DRANK  were  significant  in  accounting  for  score  variance. 
Details  of  these  interactions  are  shown  in  Table  11  and 
Figure  3.   Pilots  stationed  at  location  B  were  observed  to 
achieve  nearly  idential  mean  scores  on  the  NATOPS  closed- 
book  examinations  regardless  of  the  type  of  training  received 
while  at  VP-31  or  their  current  rank.   However,  the  mean 
scores  achieved  by  pilots  stationed  at  location  A  varied 
considerably  by  the  type  of  training  received  while  at  VP-31; 
the  non-ISD-trained  group  had  a  higher  mean  score  than  the 
ISD-trained  group.   The  difference  in  these  two  means,  as 
measured  with  a  t  test,  was  significant  at  the  .01  level. 
Additionally,  although  the  DRANK  variable  made  no  difference 
in  the  mean  closed-book  score  at  location  B,  it  did  account 
for  a  meaningful  difference  in  mean  scores  at  location  A  in 
that  the  pilots  with  higher  rank,  and  presumably  more  exper- 
ience, made  better  scores. 


E.   MULTIVARIATE  ANALYSIS  OF  VARIANCE  OF  OPEN-  AND  CLOSED- 
BOOK  NATOPS  SCORES  (TABLE  12) 

A  multivariate  analysis  of  open-  and  closed-book  NATOPS 
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Table  10 

Factorial  Analysis  of  Variance  of  Closed-Book 
NATOPS  Scores  by  Demographic  Variables 


DEMOGRAPHIC  VARIABLE 


D/F^ 


F  VALUE 


SIGNIFICANCE 
OF  F 


ISD 

DMONTHS 
DSQD 
DRANK 


1 

3.051 

.08 

2 

3.032 

.05 

1 

.177 

.67 

1 

16.271 

.00 

Two  Way  Interactions 


ISD 

DMONTHS 

2 

.051 

.95 

ISD 

DSQD 

1 

4.618 

.03 

ISD 

DRANK 

1 

1.806 

.18 

DMONTHS 

DSQD 

2 

.391 

.68 

DMONTHS 

DRANK 

2 

.598 

.55 

DSQD 

DRANK 

1 

7.842 

.01 

Total  D/F  =  122 
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Table  11 

Mean  Closed-Book  NATOPS  Scores  for  Location 
(DSQD)  by  Training  (ISD)  and  Rank  (DRANK) 


VARIABLE  VALUE  LOCATION 


B 


ISD  0  3.67  3.54 

1  3.48  3.53 

DRANK        1  3.50  3.54 

2  3.83  3.52 
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Score 


4.0 


3.0 


2.0 


(a) 


Location  A 
.Location  b 


1  2 

DRANK 


Score 


4.0 


3.0 


2.0 


vocation   B 
Location  A 


(b) 


ISD 


Figure  3.   Interactive  Effects  of  Location  (DSQD) 
with  (a)  DRANK  and  (b)  ISD  on  Closed- 
Book  NATOPS  Examination  Scores 
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Table  12 

Multivariate  Analysis  of  Variance  of  Annual  NATOPS 
Open-  and  Closed-Book  Examination  Scores 


HYPOTHESIS  OF  NO  OVERALL 

EFFECT  BY:  F*  SIGNIFICANCE  OF  F 

ISD  2.05  .13 

DRANK  4.6  7  .01 

DMONTHS  2.53  -.04 

DDESIGN  0.63  .54 

DSQD  3.90  .02 

*F  Determined  by  Wilk's  criterion  where: 

F=  i-yp    x  (NE+Q-P-1)  with  2P  and  2  (NE+Q-P-1)  d/f 

Vl 

.  _,  DET(E) 
Lt    — 


DET  (H+E) 


H  =  sum  of  squares  and  cross  product  matrix  for  the  null 
hypothesis 

E  =  error  sum  of  squares  and  cross  product  matrix 

P  =  number  of  dependent  variables 

Q  =  degree  of  freedom  of  null  hypothesis 

NE  =  D/F  of  error 
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examination  scores,  considered  simultaneously,  identified 
the  DRANK,  DMONTHS,  and  DSQD  variables  as  significant  in 
accounting  for  score  heterogeneity.   Because  of  the  previous- 
ly discussed  sensitivity  of  open-  and  closed-book  NATOPS 
scores  to  the  experience  variables  (DPANK  and  DMONTHS)  and 
the  squadron  location  variable  (DSQD) ,  the  multivariate 
analysis  was  performed  to  test  the  power  of  these  effects 
on  the  combined  scores.   Although  the  correlation  between 
open-  and  closed-book  scores  was  computed  to  be  only  .33, 
the  effects  of  squadron  location  and  both  experience  vari- 
ables were  again  statistically  significant  in  the  multi- 
variate analysis.   All  interactive  effects  were  excluded 
from  this  test  because  of  a  singular  matrix  caused  by  missing 
data  points  for  the  DDESIG  variable  which  prohibited  the 
computation  of  two-way  and  higher  interactions.   As  in  the 
factorial  analyses,  the  ISD  variable  was  not  statistically 
significant. 

F.   OPERATING  DATA  COMPARISON  (TABLE  13) 

The  most  noticeable  changes  in  operational  factors  over 
the  time  frame  examined  (FY77  through  FY79)  have  occurred 
in  the  following  four  areas: 

1.  Student/instructor  ratio 

2.  Second  tour  pilot  load 

3.  Ground  school  time 

4.  Enlisted  staff  size 
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Table  13 


A  Comparison  of  Operating  Data  for  VP-31  Training 


FACTOR 


TYPE  OF  FACTOR    FY77 


FY78 


FY79 


Student  Instructor 
Ratio 


1.18 


1.16 


1.55 


Aircraft  Flights 
per  Student 

First  Tour 
Second  Tour 

8 
6 

5 
5 

Simulator  Periods 
per  Student 

First  Tour 
Second  Tour 

0 
0 

7 
6 

Allotted  Ground 
School  Time  (Hrs) 

First  Tour 
Second  Tour 

196 
162 

232 

207 

Syllabus  Length 

First  Tour 
Second  Tour 

9 
3 
8 
2 

wks 
days 
wks 
days 

10 

6 
6 

wks 

wks 
days 
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As  illustrated  in  Table  13,  the  student/instructor  ratio  was 
increased  between  the  pre-ISD  period  of  FY77  and  FY78  and  the 
ISD  period  of  FY79  by  approximately  .39.   This  increase 
occurred  concurrently  with  an  expansion  of  the  second-tour 
pilot  load  and  would  indicate  that  the  efficiency  of  self- 
instruction  associated  with  ISD  was  being  realized.   Ground 
school  time  specified  for  the  familiarization  phase  was 
increased  between  FY77  and  FY79  due  to  the  introduction  of 
the  flight  simulator  and  the  inclusion  of  simulator  time  in 
ground  school  totals.   In  aggregate,  operational  data  seem 
to  have  remained  essentially  constant  over  the  time  frame 
examined,  except  for  a  modest,  but  steady,  increase  in  the 
enlisted  staff  size. 
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IV.   DISCUSSION 

The  preceding  analysis  suggests  that  VP-31  pilot  training 
under  the  new  ISD  condition  serves  to  account  only  for  lower 
scores  on  force-level,  closed-book,  NATOPS  examinations  by 
pilots  assigned  to  one,  of  two,  west-coast  P-3  bases.   Further, 
this  study  identified  rank,  squadron  experience,  and  squadron 
location  as  statistically  significant  effects  in  accounting 
for  the  variance  of  examination  scores.   These  results  raise 
three  salient  questions  regarding  P-3  pilot  training  and  sub- 
sequent NATOPS  evaluation. 

1.  Is  the  pilot  training  provided  by  VP-31  in  conformance 
with  the  best  and  proper  application  of  ISD? 

2.  Is  the  force-level  NATOPS  testing  program  an  ap- 
propriate measure  of  the  effectiveness  of  VP-31 
pilot  training? 

3.  Is  ISD  a  more  effective  system  for  teaching  pilots 
to  learn,  retain,  and  apply  a  knowledge  of  P-3 
operating  procedures  and  systems  than  the  previous 
traditional  system? 

To  determine  the  degree  of  conformance  between  current 

training  at  VP-31  and  the  best  and  proper  application  of  ISD 

would  require  a  formative  evaluation,  the  magnitude  of  which 

is  beyond  the  scope  of  this  study.   On  the  surface  however, 

it  appears  that  with  but  a  few  exceptions,  such  as  the  lack 

of  entry  level  testing  and  a  true  self-paced  learning  program 

which  would  allow  students  to  complete  training  at  a  self 

determined  schedule,  the  basic  principles  of  ISD  are  being 

followed.   Courseware  Incorporated,  a  private-sector  ISD 
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consultant,  has  been  largely  responsible  for  the  design  of 
VP-31's  new  training  program. 

The  appropriateness  of  the  force-level  NATOPS  testing 
program  as  a  measure  of  VP-31  pilot  training  effectiveness 
was  established  by  the  close  agreement  between  course  objec- 
tives and  the  areas  specified  for  evaluation  in  the  NATOPS 
program.   (See  Appendix  A  for  a  detailed  listing.)   Although 
a  certain  degree  of  latitude  is  permitted  in  the  construction 
of  individual  examination  questions,  the  specific  subject 
matter  is  clearly  defined  for  both  open-  and  closed-book 
examinations.   The  open-book  examination  is  a  particularly 
appropriate  measure  of  training  effectiveness  because  in 
general  it  requires  pilots  to  compute  values  associated  with 
aircraft  performance  which  are  identical  to  those  required  on 
the  job.   Analysis  of  the  variance  of  open-book  examination 
scores  suggests  that  among  the  variables  considered,  the 
effects  of  experience  (rank  and  months)  and  squadron  location 
are  statistically  significant.   Further,  the  effect  of  the 
training  variable  (ISD  or  non-ISD)  was  not  significant. 

Closed-book  examinations  measure  primarily  verbal  infor- 
mation and  intellectual  skills  associated  with  operating 
procedures  and  aircraft  systems.   Since  the  examination  re- 
quires pilots  to  first  read  and  interpret  a  question  before 
selecting  a  written  answer,  the  relationship  to  on-the-job 
requirements  is  not  as  close  as  in  the  open-book  examination. 
Analysis  of  the  closed-book  examination  scores  revealed,  as 
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with  open-book  scores,  a  sensitivity  to  the  rank  and  months 
variables.   The  type  of  training  received  was  significant  at 
the  .08  level  as  a  main  effect  and  at  the  .03  level  when 
interacted  with  the  squadron  location  variable  (ISD  trained 
pilots  scored  lower  than  traditionally  trained  pilots  at 
location  A) .   The  direction  and  consistency  of  the  main  effect 
is  very  obvious  in  Table  4.   Of  the  14  contrasts,  in  all  but 
one  the  ISD-trained  pilots  have  lower  closed-book  examination 
scores . 

The  preceding  discussion  suggests  that  organizational 
climate,  as  well  as  experience,  is  strongly  related  to  pilot 
performance  on  NATOPS  examinations.   Formal  syllabus  training 
received  at  VP-31,  either  ISD  or  traditional,  serves  to 
provide  pilots  with  a  basic  knowledge  of  P-3  systems  and 
procedures.   Retention  of  these  skills  and  further  learning 
takes  place  in  the  field  at  the  squadron  level  where  previous- 
ly learned  skills  have  the  opportunity  to  grow  or  deteriorate. 
Under  a  positive  climate,  learning  is  reinforced  and  perfor- 
mance is  rewarded.   Measurement  of  the  learning  climate 
associated  with  an  organization  should  be  based  upon  the 
attitudes  of  individuals  which  result  from  choices,  such  as 
whether  or  not  to  maintain  previous  learning  and  further 
develop  aviator  skills. 

The  ISD  method  focuses  sharply  upon  the  development  of  a 
precise  list  of  learning  objectives  based  upon  task  analyses. 
Although  the  logic  of  the  atomized  approach  toward  learning 
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specified  in  design  manuals  is  sound,  course  designers  must 
exercise  particular  attention  to  detail  when  assembling  task 
inventories  and  subsequent  learning  objectives.   If  graduates 
are  deficient  in  an  area  not  listed  among  the  objectives,  the 
problem  is  with  the  list  and  not  with  the  program.   The 
learning  of  attitudes,  which  is  not  emphasized  within  the 
Navy's  ISD  design  manual,  should  be  considered  by  course 
designers  as  a  desired  outcome  of  pilot  training. 

This  study  seems  to  indicate  that  perhaps  a  truly  positive 
attitude  toward  the  further  development  of  knowledge  in  the 
squadron  environment  is  not  being  learned  as  well  under  the 
ISD  system  as  it  was  under  the  traditional  methods.   Something 
may  have  been  overlooked  in  the  time-consuming  atomization 
process  of  ISD  when  learning  became  limited  to  those  items 
on  the  list.   Additionally,  emphasis  on  the  individualization 
of  instruction  and  self-instruction  may  have  served  to 
eliminate  the  positive  effects  of  peer  and  instructor  inter- 
actions in  the  learning  of  favorable  attitudes. 
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V.   CONCLUSION 

This  study  suggests  that  by  utilizing  force-level  NATOPS 
examination  scores  as  criteria  for  the  effectiveness  of  pilot 
training  conducted  at  VP-31,  enhancement  by  the  application 
of  ISD  has  not  yet  been  achieved.   The  following  findings  are 
germane: 

1.  A  reduction  in  closed-book  NATOPS  examination  scores 
associated  with  ISD  trained  pilots,  particularly  at  one  of 
two  west-coast  P-3  bases,  was  identified.   Although  these  ISD 
trained  pilots  did  score  lower  on  average  than  their  tradition- 
ally trained  counterparts,  it  is  somewhat  presumptuous  to 
attribute  the  difference  to  the  process  of  ISD  until  a  forma- 
tive evaluation  is  conducted  at  VP-31  to  establish  the  degree 
to  which  the  pilot- training  program  conforms  to  the  best  and 
proper  utilization  of  ISD. 

2.  Upon  completion  of  a  formative  evaluation,  further 
summative  evaluation,  under  the  controlled  conditions  of 
parallel  training  previously  discussed,  is  required  to 
determine  the  effectiveness  of  a  program  verified  to  be  a 
proper  application  of  ISD.   An  evaluation  of  this  type  is 
necessary  to  justify  the  extensive  cost  in  terms  of  manpower 
and  money  allocated  to  the  development  of  ISD  at  VP-31. 

3.  Pilot  training  provided  by  VP-31  during  the  time 
frame  examined  (either  ISD  or  non-ISD)  seems  to  have  a 
limited  effect  upon  force-level  NATOPS  examination  scores. 
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However,  the  effects  of  squadron  location  and  experience  (rank 
and  months)  are  statistically  significant  in  accounting  for 
the  variance  of  NATOPS  scores. 
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APPENDIX  A 

A  COMPARISON  OF  TRAINING  OBJECTIVES  AND  THE  SUBJECT 
MATTER  OF  NATOPS  EXAMINATIONS 


Examination  Subject 


Training  Objective 


Unit   Lesson  # 


Open 


Ordnance/Zero  Fuel 
Weight  Computation 


Fuel  Density/Minimum 
Fuel  for  Flight 

Center  of  Gravity 
Limits 


Minimum  Control  Speed 
Runway  Wind  Components 

Refusal  Speed 
VR,  RCR 

Vro,  Vlof,  V50  (3) 

Distance  to  Liftoff 


Stall  Speed/Stall 
Buffet 

Climb  Performance 


Altitude  Selection 


Max  Range  Airspeed 


Max  Range  SHP 


Book  Examination 

Define  and  Compute 
Zero  Fuel  Weight 

1 

8 

Utilize  Minimum  Fuel 
Flight  Chart 

1 

8 

Define  and  Compute 
Center  of  Gravity, 
State  Limits 

1 

8 

Compute  VMC 

3 

1 

Compute   Runway  Wind 
Component 

3 

1 

Compute  Refusal  Speed 

3 

1 

Compute  Correction  to 

3 

1 

Refusal  Speed  for  RCR 

Compute  Take  Off  Speeds 
Vro,  Vlof,  V50  (3&4  Eng) 

Compute  distance  to 
Liftoff 

Compute  Stall  Speed 


Compute  3/4  Eng  Climb- 
out  Flight  Path 

Compute  Max  Range 
Altitude 

Compute  Max  Range  Air- 
speed 

Compute  Max  Range  SHP 


1 
1 
2 
4 
6 
6 


55 


Examination  Subject 

Max  Range  Fuel 
Planning 

Loiter  Time  Predic- 
tion 

Angle  of  Attack 

Composite  Cruise 
Flight  Performance 
Chart 

Ditch  Speed 

Available  Range 
Remaining 

Landing  Ground  Roll 
Distance 

NATOPS  Manual/ 
Evaluations 

Emergencies 


Training  Objective 


Unit   Lesson  # 


Utilize  Max  Range 
Charts 

3 

4 

Compute  Loiter  Time 
Available 

3 

5 

Define  AOA 

1 

25 

Utilize  Composite 
Cruise  Performance 
Chart 

3 

7 

Compute  Ditch  Speed 

4 

12 

Compute  Available  Range 
Remaining 

3 

8 

Compute  Landing  Ground 
Roll 

3 

3 

NATOPS  Program  Review 

1 

2 

Emergencies  covered  in 


all 


Closed  Book  Examination 


Aircraft  General 
Power  Plants 

Propeller 

Electrical  System 

Hydraulic  System 

Auxiliary  Power  Unit 


General  Aircraft 

1 

Describe  the  Operation 
and  Limits  of  the 
Engines 

1 

9 

Describe  the  Operation 
and  Limits  of  the 
Propeller 

1 

15, 
17 

16, 

Describe  the  Operation 
and  Limits  of  the 

1 

3, 

5 

4, 

Electrical  System 

Describe  the  Operation 
and  Limits  of  the 
Hydraulic  System 

Describe  the  Operation 
and  Limits  of  the  APU 


13 
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Examination  Subject 


Training  Objective 


Unit  Lesson  # 


Flight  Controls 


Automatic  Flight 
Control  System 


Air  Conditioning  and 
Pressurization 


Fuel  System 

Radio  and  Nav  Aids 

All  Weather  Systems 

Armament 
Instruments 

Weight  and  Balance 


Aircraft  Performance 
and  Cruise  Control 

Ditching  and  Bailout 


Normal  and  Emergency 
Operating  Procedures 

Limitations 


Describe  the  Operation    1 
of  the  Flight  Controls 

Describe  the  Operation    1 
and  Limits  of  the 
Automatic  Flight  Control 
System 

Describe  the  Operation    1 
and  Limits  of  the  Air 
Conditioning  and 
Pressurization  System 

Describe  the  Operation    1 
and  Limits  of  the  Fuel 
System 

Describe  the  Operation    1 
of  the  Radios  and  Nav 
Aids 

Describe  the  Operation    1 
and  Limits  of  the  Foul 
Weather  Systems 

Covered  in  Tactics  Section 

Describe  the  function     1 
and  limits  of  cockpit 
instruments 

Define  and  Compute  Weight  1 
and  Balance  Data 

Compute  Max  Range  Cruise   3 


Describe  Procedures  for    4 
Ditch  and  Bailout 

Normal  and  Emergency     2&4 
Operating  Procedures 

Normal  and  Emergency     2&4 
Operating  Procedures 


6 
26,27 


18,19 


11,14 


21,22, 
23,24, 
25 

20 


25 


6 
12 
all 

all 
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